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(54) Method of speech recognition with compensation for both channel distortion and 
background noise 



(57) A method of speech recognition with compen- 
sation is provided by modifying HMM models trained on 
clean speech with cepstral mean normalization. For 
each speech utterance the MFCC vector is calculated 
forthe clean speech database. This mean MFCC is add- 



ed to the original models. An estimate of the background 
noise is determined for a given speech utterance.- The 
model mean vectors adapted to the noise are deter- 
mined. The mean vector over the noisy speech space 
is determined and this is removed from the model mean 
vectors adapted to noise to get the target model. 
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Description 

Field of Invention 

5 [0001 ] This invention relates to speech recognition and more particularly to compensation for both background noise 
and channel distortion. 

Background of Invention 

10 [0002] A speech recognizer trained with relatively a quiet office environment speech data and operating in a mobile 
environment may fail due to at least to the tow distortion sources of back ground noise and microphone changes. The 
background noise may be from a computer fan, car engine, and/or road noise. The microphone changer may be due 
to the quality of the microphone, hand-held or hands-free and, a position to the mouth. In mobile application of speech 
recognition, both the microphone conditioner and background noise are subject to change. 

15 [0003] Cepstral Mean Normalization (CMN) removes utterance mean and is a simple and effective way of dealing 
with convolutive distortion such as telephone channel distortion. See "Effectiveness of Linear Prediction Characteristics 
of the Speech Wave for Automatic Speaker 

Identification and Verification" of B. Atal in Journal of Acoustics Society of America, Vol. 55: 1304-1312, 1974. Spectral 
Subtraction (SS) reduces background noise in the feature space. See article "Suppression of Acoustic Noise in Speech 

20 Using Spectral Subtraction" of S.R Boll in IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-27 
(2): 113-129, April 1979. Parallel Model Combination (PMC) gives an approximation of speech models in noisy condition 
from noise-free speech models and noise estimates. See "An Improved Approach to the Hidden Markov Model De- 
composition of Speech and Noise" of M.J. F. Glaes and S. Young in Proceedings of IEEE International Conference on 
Acoustics, Speech and Signal Processing, Volume 1 , pages 233-236, U.S.A., April 1 992. The techniques do not require 

25 any training data. 

[0004] Joint compensation of additional noise and convolutive noise can be achieved by the introduction of a channel 
model and a noise model. A spectral bias for additive noise and a cepstral bias for convolutive noise are introduced 
in an article by M. Afify, Y. Gong, and J. P. Haton. This article is entitled "A General Joint Additive and Convolutive Bias 
Compensation Approach Applied to Noisy Lombard Speech Recognition" in IEEE Trans, on Speech and Audio Process- 

36 ing, 6(6): 524-538, November 1 998. The five biases can be calculated by application of Expectation Maximization (EM) 
in both spectral and convolutive domains. A procedure by J.L. Gauvain, et al, is presented to calculate convolutive 
component, which requires rescanning of training data. See J.L. Gauvain, L. Lamel, M. Adda-Decker, and D. Matrouf 
entitled "Developments in Continuous Speech Dictation using the AR PA NAB News Task." In Proc. of IEEE International 
Conference on Acoustics, Speech, and Signal Processing, pages 73-76, Detroit, 1996. Solution of the convolutive 

35 component by steepest descent method has also been reported. See Y. Minami and S. Furui entitled "A Maximum 
Likelihood Procedure for a Universal Adaptation Method Based on HMM Composition." See Proc. of IEEE International 
Conference on Acoustics, Speech and Signal Processing, pages 129-132, Detroit, 1995. A method by Y. Minami and 
S. Furui needs additional universal speech models, and redestination of channel distortion with the universal models 
when channel changes. See Y. Minami and S. Furui entitled "Adaptation Method Based on HMM Composition and EM 

40 Algorithm" in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, pages 
327-330, Atlanta 1996. 

[0005] The techniques presented by M.F.J. Gales in "PMC for Speech Recognition in Additive and Convolutional 
Noise," Technical Report TR-154, CUED/F-INFENG, December 1993 needs two passes of the test utterance, e.g., 
parameter estimation followed by recognition, several transformations between cepstral and spectral domains, and a 

45 Gaussian mixture model for clean speech. 

[0006] Alternatively, the nonlinear changes of both type of distortions can be approximated by linear equations, as- 
suming that the changes are small. Jacobian approach, which models speech model parameter changes as the product 
of a jacobian matrix and the difference in noisy conditions, and statistical linear approximation are along this direction. 
See S. Sagayama, Y. Yamaguchi, and S. Takahashi entitled "Jacobian Adaptation of Noisy Speech Models," in Pro- 

50 ceedings of IEEE Automatic Speech Recognition Workshop, pages 396-403, Santa Barbara, CA, USA, December 
1997. IEEE Signal Processing Society. Also see "Statistical Linear Approximation for Environment Compensation" of 
N.S. Kim, IEEE Signal Processing Letters, 5(1): 8-10, January 1998. 

[0007] Maximum Likelihood Linear Regression (MLLR) transforms HMM parameters to match the distortion factors. 
See "Maximum Likelihood Linear Regression for Speaker Adaptation of Continuous Density HMMs" by C.J. Leggetter 
55 and P.C. Woodland in Computer, Speech and Language, 9(2): 1 71 -1 85, 1 995. This method is effective for both sources 
but requires training data and introduces the dependence to the speakers. 
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Summary of Invention 

[0008] In accordance with one establishment of the present inventor a new method that Handles simultaneously 
noise and channel distortions to make a speaker independent system robust to a wide variety of noises and channel 
5 distortions. 

Description of Drawing 

[0009] 

w 

Figure 1 illustrates a speech recognizer according to one embodiment of the present invention; and 

Figure 2 illustrates the method of the present invention generating 

15 Description of Preferred Embodiments of the Present Inventions 

[0010] Referring to Fig. 1 there is illustrated a speech recognizer according to the present invention. The speech is 
applied to recognizer 11. The speech is compared to Hidden Markov Models (HMM) 13 to recognize the text. The 
models initially provided on those with speech recorded in a quiet environment and the microphone of good quality. 

20 We want to develop a speech model set suitable for operating in the simultaneous presence of channel/microphone 
distortion and background noise. In accordance with the present invention, a speech model set is provided using 
statistics about the noise and speech. A low computation cost method integrates both PMC and CMN. 
[0011] Referring to Figure 2, the first Step 1 is to start with HMM models trained on clean speech, with cepstral mean 
normalization. We modify these models to get models to compensate for channel/microphone distortion (convolutive 

25 distortion) and simultaneous background noise (additive distortion). For an HMM model, we have a lot of parameter 
but only change one subset of the parameters and that is mean vectors rn pJk . The mean vectors m pJ k of the original 
model space is modified where p is the index of the Probability Density Function (PDF), j is the state and k is the mixing 
component. 

[0012] The second Step 2 is to calculate which is the mean mel-scaled cesptrum coefficients (MFCC) vector over 
30 the trained database. Scan all data and calculate the mean to get b. 

[001 3] The third Step 3 is to add mean b to each of this mean vector pool represented by m p j k equation (1 ) to get: 

35 

For example, there could be 100 PDFs, 3 states per PDF and 2 vectors per state, or a total of 600 vectors. 
[001 4] The fourth) Step 4 js for a given inDuyest utterance, an estimate of the background noise vectorx is calculated. 
[0015] Let u'A [ ,.. ] r and v r A [ ,.. ] r : where 1 means that the values are represented in log-spectral 
domain. 1 2 1 2 

40 [001 6] We introduce the combination operator © such that: 

w' Au' ®v f =1^, ^J wD ] T (2) ' 

45 with 

( v/ =log(exp( / M/ ) + exp( / l7 .)) (3) 
50 [0017] In Step 5, we calculate the mean vectors adapted to the noise X using equation 4. 



m Ptj , k = IDFT{DFT(m Ptj , k ) © DFT( X ) ) . 

55 

(4) 
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where DFT and IDFT are, respectively, the DFT and inverse DFT operation, rn p j k is the noise compensated mean 
vector. 

[0018] Equation 4 involves several operators. DFT is Discrete Fourier Transform and IDFT is the Inverse Discrete 
Fourier Transform. The 0 is an operation with two vectors. A © B = C. How ® is defined, we look at equations 2 and 

5 3^ Equation 2 says the operation + operates on two vectors u and v and the result is a vector of D dimension or [ w , 
w w ] T where T is the transposition. We take the two vectors and produce another vector. We need to specify eadh 
efemeift in the resultant vector. Equation 3 says that the jth element in that vector ( w ) is defined by the exponential of 
the element of u added to the exponential if the jth element of v and take the log of tKe combination of the exponential 
of u added to the exponential of the j the element of v. This completes the definition of Equation 4. 

w [001 9] In the following steps, we need to remove the mean vector b of the noisy data y over the noisy speech space 
N (from the resultant model). One may be able to synthesize enough noisy data from compensated models but this 
requires a lot of calculation. In accordance with the present invention the vector is calculated using statistics of the 
noisy models. The whole recognizer will operate with CMN (cepstral mean normalization mode), but the models in 
Equation 4 are no longer mean normalized. We have dealt with additive noise. The second half of the processing is 

15 removing the cepstral mean of our models defined in Equation 4. This is not difficult because we have the models in 
Equation 4. In Step 6, we need to integrate all the samples generated by Equation 4 to get the mean. Mean is b. 
Equation 5 is this integration. 

[0020] Let , be the variable denoting PDF index, J be the variable for state index, and K be the variable for mixing 
component index. 

20 

b = E{y} 

= N y 2. p ZjL k P, (p) PJ, (y,p) P K|l ,j (*\P'j) py,„ J, K (y|P« t *> dy (5) 

25 

Since 

Piy\p,j,k) = N(y, IDFT(DFT(m Pf1fk ) © DFT(5t)), B P/j/k ) (6) 

30 

We have 

35 b = XpLjX, P, (p)PJ, {fl p)p K „ j (k\p,j) & pJk (7) 

A 

Equation 7 shows that b can be worked out analytically, and it is not necessary to do the physically generation and 
integration. The final result is Equation 7 which is the integration into several sums. Sums over probability density 
functions and the sum over states and sum over mixing components. Then you have several quantities. The P H is the 
40 probability of having the PDF index. The Pj given H is the probability of being in the state if given the PDFp. The next 
is the probability the mixing component p^j given we have the PDF index. The mean vector of the compensated mode. 
To make this complete we remove this b from the compensated model to get the target model. This is Step 7. The 
target model is: 

45 A a 

m P j.* = rn P j,k ' b ( 8 > 

This is what we want to^load into our recognizer. This operation is done for each utterance. 

[0021] Calculation of b thus requires the knowledge of probability of each PDF. There are two issues with P (, = p) : 

50 

• It needs additional storage space. 

• It is dependent of the recognition task e.g. vocabulary, grammar. 

55 [0022] Although it is possible to obtainUiat probability, we want to consider the following simplified cases. 

[0023] This operation to calculate the b can be simplified with three approximations. The first one uses equal prob- 
abilities for P H (p) or constraint C. 
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1 . Use equal probabilities for P H (p) 



P H (p) = C 



(9) 



10 



2. Use equal probabilities for P H (p), Pj\.. P(flp) and P M d (k\h,j). 

P H (p)=C 



(10) 



15 



20 



3. In fact, the case described in Eq-10 consists in averaging the compensated mean vectors m p j k . Referring to 
Eq-4 and Eq-1 , it can be expected that the averaging reduces the speech part m p j - k just as CMN does. Therefore, 
Eq-7 could be further simplified into: 



25 



b = IDFT ( DFT ( b ) 



DFT ( X ) ) 



(11) 



30 



35 



40 



45 



50 



55 



The model m T p j kt is then used with CMN on noisy speech. Unfortunately, b is a function of both channel and 
background noise in all above cases. In other words, in presence of noise, there is no guarantee that the channel 
will be removed by such a vector, as is for CMN. 

[0024] A subset of WAVES database containing hands-free recordings was used, which consists of three recording 
sessions: parkedtrn (car parked, engine off), parked (car parked, engine off), and city-driving (car driven on a stop and 
go basis). 

[0025] In each session, 20 speakers (10 male) read 40 sentences each, giving 800 utterances. Each sentence is 
either 1 0, 7 or 4 digit sequence, with equal probabilities. The database is sampled at 8kHz, with MFCC analysis frame 
rate of 20ms. Feature vector consists of 10 statis and 10 dynamic coefficients. 

[0026] HMMs used in all experiments are trained in TIDIGITS clean speech data. Utterance-based cepstral mean 
normalization is used. The HMMs contain 1 957 mean vectors, and 270 diagonal variances. Evaluated on TIGIDIT test 
set, the recognizer gives 0.36% word error rate. 

[0027] To improve the performance in noisy environment, the variances of the Gaussian PDFs can be MAP adapted 
to some slightly noisy data, e.g. WAVES parked_eval data. Such adaptation will not affect recognition of clean speech, 
but will reduce variance mismatch between HMMs the noisy speech. The new algorithm is referred to as JAC (joint 
compensation of additive noise and convolutive distortion). 

Table 1 : 



word error rate (WER) (%) as function of driving conditions and compensations methods 




PARKED 


DRIVING 


BASELINE 


1.38 


30.3 


CMN 


0.59 


18.8 


PMC 


1.74 


6.29 


JAC FULL PROB (Eq-7) 


0.84 


1.93 


JAC CONST PDF PROB (Eq- 9) 


0.84 


2.00 




1.16 


2.18 


JAC AVERAGE (Eq-10) 


0.48 


1.91 


JAC SIMPLIFIED (Eq-11) 
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[0028] Table- 1 shows that: 

• Compared to noise-free recognition (WER) (0.36%), without any compensation (BASELENE) the recognition per- 
formance degrades severely. 

5 

• CMN effectively reduces the WER for parked data, but is not effective for driving conditions where additive noise 
becomes dominant. 

• PMC substantially reduces the WER for driving conditions, but gives poor results for parked data where microphone 
10 mismatch is dominant. 

• All JAC cases give lower WER than non-JAC methods. 

• Simplifying Eq-7 to Eq-9 then to Eq-10 results in progressive increase in WER, although the degradation is not 
15 severe. Especially, information in PDF probability is not critical to the performance. 

• Simplified JAC gives lowest WER in all tests. For this hands-free speech recognition, the new met reduces word 
error rate by 61% for parked condition and 94% for city driving condition. 



Claims 

1 . A method of modifying HMM models trained on clean speech with cepstral mean normalization to provide models 
that compensate for simultaneous channel/microphone distortion and background noise (additive distortion) com- 

25 prising the steps of: 

A 

for each speech utterance calculating the mean mel-scaled cepstrum coefficients (MFCC) vector b over the 
clean database; 

adding the mean MFCC vector b to the mean vectors m^ of the original HMM models where p is the index 

30 of PDF, j is the state, and k the mixing component to get in m p j k ; 

for a given speech utterance calculating an estimate of the background noise vectorx_; 
calculating the model mean vectors adapted to the noise x using m p y^= ID FT (DFT (m p j tk ® DFT (x)) to get 
the noise compensated mean vector where the Inverse Discrete Fourier Transform is taken sum of the Discrete 
Fourier Transform of the mean vectors m p j k modified by the mean MFCC vector b added to the Discrete 

35 Fourier Transform of the estimated noise x I and 

calculating the mean vector b of the noisy data over the noisy speech space, and removing the mean vector 
b of the noisy data from the model mean vectors adapted to noise to get the target model. 

A 

2. The method of Claim 1 wherein the step of calculating the mean vector b of the noisy data over the noisy speech 
40 space will calculate the vector using statistics of noisy model using : 

b = Z p XjZ k P, (p)P Jl (j\P) P Kl „ J (*|P^ ™ P j,k 

45 where H is the variable denoting PDF Index J is the variable for the sate index and K is the variable for mixing 

component index 

A 

3. The method of Claim 2 wherein said calculating the mean vector b uses equal probabilities for P H (p) 

50 

p H (p) = C. 

4. The method of Claim 2 wherein equal probabilities for P H (p), Pj , P(fip) and P w ,>j (Mh,j) is used. 
55 P H (P) = C 
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Pj >H (flp) = 0 
P K \h.A m P>J) = e 

A 

5. The method of Claim 3 wherein mean vector b becomes equal to: 



b = IDFT ( DFT ( b ) © DFT (X)). 

6. . A method of speech recognition with compensation for channel distortion and background noise comprising the 

steps of: 

providing HMM models trained on clean speech with cepstral mean normalization; and modifying said models 
by the method of any preceeding claim 

7. A speech recognizer with compensation for channel distortion and background noise comprising in combination: 

adapted HMM models generated by modifying HMM models trained on clean speech with cepstral mean nor- 
malization wherein said models are adapted by: 

for each utterance: 

calculating the calculating the mean mel-scaled cepstrum coefficients (M FCC) vector b over the clean 
database; 

A 

adding the mean MFCC vector b to the mean vectors rr\ pJk of the original HMM models where p is 
the index of PDF, j is the state, and k the mixing component to get in m p j k ; 

for a given speech utterance calculating an estimate of the background noise vector x ; 

calculating the model mean vectors adapted to the noise x using 



m p , jfk = IDFT (DFT (rn p , jfk © DFT (X)) 

to get the noise compensated mean vector where the Inverse Discrete Fourier Transform is taken 
40 sum of the Discrete Fourier Transform of the mean vectors modified by the mean MFCC vector 

b added to the Discrete Fourier Transform of the estimated noise x; and 

calculating the mean vector b of the noisy data over the noisy speech space, and removing the mean 
vector b of the noisy data from the model mean vectors adapted to noise to get the adapted model;and 
means for comparing the adapted model to the speech input utterance to recognize the input speech 



8. The recognizer of Claim 7 wherein the step of calculating the mean vector b of the noisy data over the noisy speech 
space will calculate the vector using statistics of noisy model using : 

50 £ - N y E p Z y ^ P, (p)P Jv (/|p)P K| J (*|A7) ™ pJM 

where H is the variable denoting PDF Index J is the variable for the sate index and K is the variable for mixing 
component index 6. The model of Claim 5 wherein the step of calculating the mean vector b of the noisy data over 
the noisy speech space will calculate the vector using statistics of noisy model using : 



55 



b = Z p ZjZ k P, (p) P JX (j\p)P Kl j(q P ,j) & pJk 
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where H is the variable denoting PDF I ndex J is the variable for the sate index and K is the variable for mixing 
component index 

9. The recognizer of Claim 8 wherein said calculating the mean vector b uses equal probabilities for P H (p) 

5 

P H (p) = C. 

10. The recognizer of Claim 8 wherein equal probabilities for Ph(p), Pjl P(flp) and P M j (k\h,j) is used. 

10 

P H (p) = C 



15 



A 

20 11. The method of Claim 8 wherein mean vector b becomes equal to: 

b = IDFT { DFT ( b ) © DFT (X)). 

25 

12. A method of speech recognition with simultaneous compensation for both channel/microphone distortion and back- 
ground noise comprising the steps of: 

modifying HMM models by a method as claimed in any of claims 1 to 5. 

30 



35 



40 



45 



50 
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